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INTRODUCTION 



So long as there are schools and so long as there is some 
dissatisfaction with what children study in schools, new curri- 
culum materials will bo developed. Perhaps because there are 
now so many schools in the United States and so much dissatis- 
faction with what children study in them, the development of 
now curriculum materials is proceeding today at a previously 
unprecedented pace. Amidst the flurry of often richly en- 
dowed curriculum development activity, the conviction is steadily 
gaining ground among educators that the curriculum materials 
produced are not necessarily ”good” simply because they are 
"new.” Before foisting the new products of curriculiam devel- 
opment projects on unsuspecting children, responsible educators 
are asking pertinent questions about the outcomes that can be 
expected from using the new materials and their suitability 
for different groups of students. Fortunately, the curriculum 
developers, by and large, have accepted the responsibility for 
seeking answers to such question as a part of their develop- 
mental work. 

When he considers the outcomes to be anticipated from 
students* use of his materials, two kinds of questions confront 
the curriculum developer. The first concerns the effectiveness 
of the materials in getting students to learn the particular 
subject matter that they are designed to teach. Second, and 
at least equally important, is the effect of the materials on 
the students* general perceptions of the subject or discipline 
being studied and of its modes of inquiry. The study presented 
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hoTG consid.Gi’s SLspGcfcs of both of thosG kinds of ciuGstions 
of concern to the curriculum developer. It seeks to illustrate 
some of the ways by which a relatively modest evaluative study 
can be of considerable benefit in furthering the work of a 
curriculum development project. 

One of the shortcomings of many evaluative studies of 
curriculu m materials has been that the approach has been too 
gross. Typically, total test scores are used to measure student 
achievement or pretest-posttest changes in mean scores are used 
to measure student gain, VJhile these measures are valuable and 
oj^ould be a necessary part of an evaluative study, they fail 
to provide the curriculum developer with sufficient specific 
information about the strengths and weaknesses of his materials. 
Particularly in the early phases of a curriculum development 
project, it is important to have as specific information as 
possible about what knowledge and which ideas are mastered 
successfully by the students, about where the students failed 
to attain mastery, and about the changes, if any, in students* 
perception of the subject that accompany instruction with the 
new materials. Data that will yield such specific information 
can be obtained quite readily in a carefully designed evalu- 
ative study. In fact, much of the data of this kind is 
frequently collected in the course of a study, but the data 
are seldom fully exploited in the analysis. This study 
illustrates some procedures of analysis and interpretation 
that may be utilized to yield information of direct value for 
the continuing development of curriculum materials. 
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DESCRIPTION OP THE STUDY 

Curriculum rnndorials dove loped by the University of Illinois 
Elementary School Science Project (ESSP) were the subject of 
this study. The focus of the ESSP is the development of curric- 
ulum materials in the area of astronomy - ^'materials that are 
sound astromically, that reflect the structure of the subject 
as it is viewed by astronomers of stature, and that can be 
handled by teachers and children in actual classrooms,”'^ The 
ESSP materials for students consist of a series of six booklets, 
richly illustrated with line drawings and containing reading 
text and many appropriately interspersed pupil activities, A 
comprehensive Teacher^ s Guide accompanies each of the student 
booklets. Charting the Universe (1963 edition). Book 1 of the 
series, was used in this study. 

Purposes of the Stu dy 

Two main purposes were conceived for this study, viz , , 

(A) to assess the effectiveness of the ESSP materials; and 

(B) to assess t5ae effect oia stu>dents of stu^d^^uag tlao TjSSP materi- 
als, (In the preceding sentence, and throughout this report, 

the term ”ESSP materials” should be understood as referring 
only to ESSP Book 1, Charting the Universe , ) Under each of these 
purposes, several related questions were given consideration, 

^ J, Myron Atkin, "Some Evaluation Problems in a Course Content 
Improvement Project, ” Journal of Research in Science Teaching , 
1, 129-132 (1963). 
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Assessment of the effectiveness of the ESSP materials in- 
cluded an attempt to ascertain how well the students learned 
the particular topics the materials wore designed to teach. The 
Teacher »s Guide (page i) states that "Book I presents a sequential 
development of ideas to show how astronomers are able to chart 
the universe," and this sequence of ideas represents the topics 
to be mastered through study of the materials. Moreover, the 
ESSP curriculum developers and the investigator believed that 
this approach to the study of astronomy would also result in 
a concomittant increase in students* general knowledge of aston- 
omy,* oven though such specific information about astronomy was 
not explicitly taught in Book 1. Hence, the assessment of 
subject matter achievement included both of these aspects. 

The hypothesis tested was: 

HYPOTHESIS 1. Study of ESSP materials will increase students* 

knowledge of astronomy and of how astronomical 
information is obtained. 

Implicit in this hypothesis, in. view of the emphasis in 
certain sections of Book 1, is that "how astronomical information 
is obtained" includes the mastery of several skills in making 
measurement. A further question related to this first hypothesis 
regarding subject-matter achievement concerned the connection 
between any such achievement and students* general scholastic 
For investigating this question, the hypothesis 
formulated was: 

HYPOTHESIS 2. Subject matter achievement is positively correla- 
ted with a student *s general scholastic ability. 

The second main purpose of this study was to assess the 
effect on students of studying the ESSP materials. Questions 
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under this head are concerned with investigating changes in 

students* perceptions of certain aspects of science and science 

study. Though the effocting of such changes in perceptions 

may not be an explicitly stated objective, the influence of 

curriculum materials on students* perceptions is an inevitable 

accompaniment of instruction. We believed it desirable and 

important to investigate this effect of the ESSP materials. 

Specifically investigated were possible changes in students* 

general understanding of science, the relationship of any such 

changes to subject-matter achievement, and possible changes in 

students* perception of astronomy, arithmetic, scientists, and 

the study of science. The pertinent hypotheses were: 

HYPOTHESIS 3, Study of ESSP materials will increase students* 

general understanding of science (as measured 
by the Test On Understanding Science ), 

HYPOTHESIS I4., A student *s gain in general understanding of 

science is positively correlated with subject- 
matter achievement, 

HYPOTHESIS 5, Study of ESSP materials will affect students* 

views of astronomy, arithmetic, scientists, and 
learning experiences in science. 



I 
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Instruments 

To obtain data for assessing the effectiveness of the 
ESSP materials, two subject-matter achievement tests were 
constructed. Some of the multiple-choice items for those tests 
wore obtained from tests used previously by the ESSP, but most 
of the test items were devised especially for this study. 

The subject-matter pretest (called "Charting the Universe Tost, 
Form 207") consisted of 28 multiple-choice items. Of these, 

15 items dealt with material specifically taught in ESSP Book 1, 
and 13 items tested for selected topics of general knowledge 
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in astronomy, (Tho Book 1 items on the subject-matter protest 
did not touch on all the material taught in Book 1; in order 
not to make the test too long and to avoid a possibly frustra- 
ting experience for the children, no questions wore included 
on Book 1 material about which none or very few of the pupils 
were expected to know prior to the study. The participating 
teacher and the investigator together decided what topics 
should not be included,) Prom the pretest administration, the 
test reliability of the total tost was found to be ,597 (Kuder- 
Richardson Formula 20), The reliability for the subtest of 
Book 1 items was *353> and the reliability of the subtest of 

General Knowledge items was ,475. 

The subject-matter pOsttest (called ^Charting the Universe 
Test, Form 208”) contained 42 items and included the 28 items 
from the pretest, nine additional multiple-choice items 
concerned with Book 1 material, and five items calling for 
the student to demonstrate his skills in making measurements 
of lines and angles as taught in Book 1, For the total test, 
the reliability computed from the posttest administration 
data was ,829, For the subtest consisting of the 28 pretest 
items the reliability was , 676 ; for the subtest of the Book 1 
pretest items, the reliability was ,5^9j for the subtest of 
General Knowledge items, the reliability was ,603; and for 
the subtest of all Book 1 items, the reliability was ,759. 

To obtain data bearing on the study ^ s second main purpose, 
to assess the effect on students of studying the ESSP materials, 
two additional instruments were administered both as pretest 
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and posttest. One of these was the Test On Understa nding Science 
(TOUS), FoiinEx, which is one of a group of instruments, de- 
veloped by the investigator and several collaborators, to 
measure students* understanding of salient aspects of the aims 
and processes of scientific enquiry, the characteristics of 
scientists, and the dynamics of the scientific enterprise. 

TOUS, Form Ex, contains 36 multiple-choice items, many of 
which call for the making of quite careful discriminations to 
select the best answer from the four alternative responses 
presented. From the pretest administration of TOUS in this 
study, the test reliability computed was .57^5 from the post- 
test administration, the reliability was .6I}.3. (The values 
for the test reliability of TOUS are somewhat lower than those 

typically found from other studies c ) 

Lastly, to complete the testing battery for this study, 
an experimental semantic differential instrument was designed. 
The semantic differential developed by Osgood and his assoc- 
iates, though hitherto little used by researchers in science 
education, provides a promising technique for assessing 
students* perceptions of concepts relevant to the teaching 
of science. In a typical semantic differential instrument, 
the student is asked to indicate his associations of a given 
concept with a series of bipolar word-pairs (e.g., good-bad, 
powerful-weak, exciting-dull). Working rapidly, he checks the 
one of five or more available positions between each pair of 
bipolar adjectives which represents how he associates these 
words with the concept. The result of the checking process is 
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a series of ratings of the given concept along a dozen or more 

adjectival bipolar scales. The same set of scales is usually 

used for rating several concepts appearing on successive pages 

2 

of the semantic differential instrument. This was the practice 
adopted in the semantic differential instrument, called "Word 
Association Study" (WAS), designed for the present study. 

The WAS instrument consisted of two cover pages containing 
an explanation of how to make responses on it and eight pages 
of 15 five-position adjective scales to be used in rating the 
following eight concepts: ASTRONOMY, ARITHlffiTIC, MOST SCIENTISTS, 

EXPLORING NEW IDEAS, DOING SCIENCE EXPERIIffiJNTS, RE.:^iDING ABOUT 
SCIENCE, MKING IvEA SURE IViE NTS, SCIENCE TEACHER. The l5 bipolar 
adjectival pairs, in the order of their appearance on each 
pag^were: quick-slow, weak-powerful, dirty-clean, hard-soft, 

important-unimportant, dull-exciting, mannish-womanish, good-bad, 
unenjoyable- enjoyable, moving- still, useless-useful, changing- 
permanent, foolish-wise, interesting-boring, easy-difficult. 

Eight concepts times 1^ scales gives a total of 120 ratings 
to be made by a student on the WAS instrument. The use of the 
WAS instr-ument in this study represents the first application, 
as far as we know, of the semantic differential technique in 
an evaluative study of elementary- school science curriculum 
materials. 



^ For further discussion of the theory and development of the 
semantic differential, see Charles E. Osgood, George J. Suci, 
and Percy H. Tannenbaum, The Measurement of Meaning , Univer- 
sity of Illinois Press, Urbana, ill., 1957. 
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Population and Procedures 



Tho students included in the study comprised the entire 
fifth grade in tho University of Chicago Laboratory School during 
tho school year 1963-6i|.. These 92 students, 43 boys and 49 
girls, were in four instructional groups of 23 students each. 

The range of I.Q. scores (Henmon- Nelson Test: Elementary Form) 

for the entire group was 88 to 179, with a median score of 

124. 

All of the groups were taught by Mss Barbara Wehr, science 
teacher in the University of Chicago Laboratory Schools. Miss 
Wehr has had more than ten years of teaching experience with 
a strong emphasis in elementary- school science. For ten weeks 
of instruction, the pupils studied the ESSP materials in Book 1, 
Charting the Universe ♦ Each group met for three 50-minute 
periods per week, A copy of tho pupil book was provided for 
each child, and the suggested equipment and supplies for all 
the pupil exercises were made available. The teacher carefully 
followed the ^'suggestions for teaching" presented in the 
Teacher*s Guide and also chose to include most of the 'Supple- 
mentary activities" and "supplementary exercises," In the 
course of the instruction, she prepared 11 sheets of additional 
exercise material for the use of the students. 

The three pretests (Charting the Universe Test, Form 207; 
TOUS; and WAS) were administered to the pupils in the four 
instructional groups on 7^ 10 > and 11 February 1964* None 
of the pretests was administered by the participating teacher. 




j 



I 













- 10 •• 

Study of tho ESSP materials began in each instructional group 
on the class meeting following the third preuost and continued 
in every succeeding class meeting during ten weeks of instruction. 
The instructional period was interrupted by one week of vacation 
and one week of no science classes during a school camping trip. 
Following the instructional period, the three posttests (Chart- 
ing the Universe Tost, Form 208; TOUS; and WAS) were administered 
on 8,9, and 10 Mrj, 

In constituting the four instructional groups at the begin- 
ning of the school year, no selection criteria had been applied 
and pupils were randomly assigned to a group. During the 
instructional period of this study, all four groups used tho 
same materials and were taught with the same procedures by the 
same teacher. Hence, tho four instructional groups were con- 
sidered to be a single population, and the individual student 
was taken as the unit of analysis. Data collected in the study 
were punched into IBM cards, and data processing was accomplished 
through the facilities of the Education Statiscics Laboratory 
and the IBM 709ij. Computation Center at the University of Chicago, 
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FINDINGS 

A. EFFECTIVENESS OF THE ESSP MTERIALS 

• , ^ 

« * 4 * • » « * 

^ Sub.ject-Matter Achievement 

In order to test the first hypothesis, that study of ESSP 
materials will increase students* knowledge of astronomy and 
of how astronomical information is obtained, a t test, was used 
comparing the scores on tho pretests with the scores on the 
same measures as posttests. The t was computed for a Total 
Astronomy Test, which was the combined scores of the General 
Knowledge and Book 1 items, and for the General Knowledge Tost 
and for the Book 1 Test separately. Because the pre and post 
scores wore obtained from the same pupils, they were presumed 
to be related, and the ;b for correlated groups was computed. 

The results are shown in Table 1. 

The posttest results, in all three cases indicate that a 
significant difference exists. More than chance factors were 
involved in tho increase, and tho greater achievement can 
probably be ascribed to the use of tho ESSP materials. 

To test tho second hypothesis, that subject matter achieve- 
ment is positively correlated with a student’s general scholas- 
tic ability, a partial correlation coefficient was derived. 
Performance on the posttest was assumed to be related to per- 
formance on the pretest. Our purpose, however, was to detect 
the correlation between posttest scores and I.Q., nullifying 
the effect on the pretest. A first order partial correlation 
provides this information. 
















I 

4 

1 



- 12 " 



TABLE 1 



Comparison of Pretest and Posttest Scores on Subject Matter Tests 

(N = 90) 













Significance 






Mean 


S.D. 


t 


Level 




pre 


5.59 


2.088 


7.63 




Book 1 


post 


7.36 


2.555 


p< , 001 




General Knowledge 


pre 


4.82 


2.12 


3.84 


p < .001 


post 


5.62 


2.41 








pre 


10.41 


3.52 


7.58 




Total 


post 


12.98 


4.47 


p <^, 001 
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Partial correlation coefficients were obtained for the 



Gonoral Knowledge items, the Book 1 items, and for a total 
score which combined the first two. (See Table 2). Trans- 
forming the partial r to a corresponding Z value, we found a 



confidence level for the value of the partial correlation 

1 



using the normal distribution of the statistic Z(-4)“^, 

3 

according to the procedure described by Hays, 

As can be seen from the table, all three correlations are 
significant. Both on items particular to the ESSP materials 
and on items measuring general knowledge about astronomy, 
achievement was related positively, thou^ only slightly, to 
general scholastic ability. 

As with any correlation coefficient, care must be exer- 

2 

cized in the interpretation. Using the relationship that r 
oQuals the proportion of the total variance of one factor 
accounted for by the other, we find that only about 7 % of the 
variance of Book 1 scores is accountable to I.Q. Only about 
of the variance of General Knowledge scores is attributable 



to I.Q. 

Intelligence tests tend to be tests of verbal ability. 
This may be what we are attempting to correlate with our 
achievement test scores. We find little relation because the 



ESSP materials seem not to demand such verbal facility, and 
success with them is more likely to bo related to other factors 



wo have not measured. 



^ William L. Kays, Statistics for Psychologists . Holt, Rinehart 
and Winston, New YorlcJ 19b3. Page 57b, 
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TABLE 2 



Partial Correlation 
Matter Posttest 


Coefficients 
Holding the 


: I.Q. with Subject- 

Pretest Constant 




I.Q. 


Significance Level 


Book 1 


.260 


p C.01 


General Knowledge 


• 294 


p <.01 


Total 


GO 

O 

• 


p< .01 




Analysis of Test Itoms 

\^Ale the analysis and comparison of whole-test moans yields 
information as to the effectiveness of the methods and/or mater- 
ials being used, closer inspection of specific items may reveal 
areas of knowledge and iinderstanding in which the materials are 

particularly successful or unsuccessful. 

A technique for this kind of analysis is MoMemar's chi 
square tost of change. A fourfold contingency table is con- 
structed for each item illustrating the numbers who had the 
item right or wrong on the protest and posttost. Chi square 

Pretest 



Wrong 



P 

o 

s 

t 

t 

e 



Right 






Right 

B 



s 

t 



Wrong ^ ^ 

is equal to (A-D)^/ (A+D) and has one degree of freedom. Using 

this statistic, we obtain a measure of the significance of tho 

change in the responses to tho item. 

Of the as test items which appear on both the pretest and 
tho posttost, 11 had significant changes. Those 11 were comprised 
of 6 of the 13 general knowledge items and 5 of the l5 Book 1 



items . 



^ Quinn McNemar, Psycholo gical Statistics. John v/iley. 
New York, 1962. Pages 224^22?. 
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B ook 1 Items ; Table 3 

Three of the Book 1 items with significant changes from 
pre to posttest are all concerned with the topic of angles and 
measurement of angles and triangles (#7 i, #14^ and #22). Not 
only did more pupils choose the right answer on the posttest, 
but also, except in the case of choice A in item #22, fewer 
p\ 3 _piXs chose each incorrect answer. Since these three items are 
the total number of items on the test measuring achievement 
of knowledge on this topic, it appears that the materials are 
quite effective in helping children learn about the measurement 
of angles and triangles. Among the pupils, ?2^ Sot item #7 
correct on the posttest; 86^ got item #ll4- correct, and 63^ got 
item #22 correct. The first two can certainly be considered 
to indicate mastery of the materials, and the latter approaches 
mastery if we consider a 'JOfo class achievement to be our 



criterion. 

Two other Book 1 questions are included in Table 3. On 
both of these items also, the change from pretest to posttest 
results in a significant chi square. But on the posttest only 
l^fo of the pupils got #21 correct; apparently the students did 
not learn to do the kind of estimating called for in this tost 
item. Only 22^ of the pupils got #28 correct. The most popular 
incorrect response to this item both on the pretest and 

the posttest {l\.2ffo) was alternative G, which names the circle 
which is the largest as drawn in the diagram. The idea of appar- 
ent angular diameter was probably not adequately mastered by 
the students. Of note also, among 92 pupils, 74 got #21 wrong 



SiittiiiaMMaaiia^ffiiaaiaiiaaaaMasiaaaBiasa^^ 
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TABLE 3 



Book 1 Items with Significant Chi Square Test of Change: 
Proportions of Responses by Students 



7 . 



One angle of a triangle is IqO degrees and another one is 70 degrees 
What must the third angle be? 



A. 

*B. 

C. 

D. 

E. 



It could be anything; triangles come in all sizes. 
70 degrees 
40 degrees 

You need to measure to find out. . • 

57‘i degrees 

2 
X 





A 


B 


C 


D 


E 


pre 


.27 


.30 


.11 


.20 . 


CO 

0 

• 


post 


CO 

0 

• 


.72 


.04 


.12 


. 04 



27.769 p<.001 





14. Of L and M above, which is the larger angle? 



A. 

B. 

C. 

•x-D. 



E. 



Angle L. It covers more of the page. 

Angle M. . 

It depends on vjhat you mean by "angle.” 

It looks like they are both the same, but you need 
to measure to be sure. 

It depends on what size circle they are in. 





A 


B 


C 


D 


E 


X-" 




pre 


. 1 ? 


.07 


.08 


.?7 


.14 


17.780 


p<.001 


post 


.05 


.05^ 


.02 


.86 


.01 







i 
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TABLE 3 (oont'd) 



I 



21 Tom made several measureraeiits with his ruler. One of the lines 
he measuT-ed ended between the mark indicating D inches and the 
raark indicating 8 l/lO inches. It was slightly nearer the b 
inch mark. \^iich of the following numbers should Tom record, 
if he wants to record the most accura.te measurement? 



A. 

C. 

D. 


8 inches. 

8.0 inches. 

8.00 inches. 

8 . 1 iriche s . 




< 




A 


B 


c 


D 


pre 


.38 


.04 


.05 


.53 


post 


.33 


.15 


.09 


.44 



X' 



5.??? .02>p>.01 



I 




A. Yes. They look very similar. 

B. Yes. The sides are almost the same size, 
c! No. They are congruent. 

'X-D. No. The angles in similar triangles must be the same. 
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TABToE 3 (cont»d) 



28, In the diagram below, which circle appears to be the largest 
when viev^ed from point P? 

A, Circle R, 

•x-B, Circle S, 

C, Circle T, 

D, They all appear to be the same size, 

E, R and T appear larger than S, 






A 


B 


C 


D 


E 




pre 


.15 


.07 


.55 


.15 


,02 


8.694 


post 


.23 


,22 


,42 


,10 


,01 





8,894 ,005>P>.001 



■ 
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on both pro and posttost, and 69 had #28 wrong both times. 

Wiilo the statistic denotes significant shift, the results for 
those two items on the posttost do not at all indicate mastery. 

In a further attempt to discern more precisely what pupils 
loarned or did not learn in the ooirse of time spent with tho 
ESSP materials, tho posttest results on Book 1 questions wore 
oxaininod for items mastered by fewer than 25^5 of tho pupils. 

(See Table 4) Three such items wore found; they are numbers 

13, 18, and 2$. 

Question #13 involves taking the idea of a scale model one 
stop further than tho way it is presented in tho ESSP materials 
Thirty-throe per cent of the pupils choose incorrect response E 
on the posttost, however, which indicates that the concept that 
a scale model involves a ratio at least was known by one third 

of tho pupils. i 

Question #l8 requires information loarned in Chapter 7. j 




As the time allotted for this study came to a close, this final 
chapter did not receive attention comparable to that for the 
other chapters. This may account for poor pupil performance on 

this item. 

An explanation for the poor performance on #2$ can be found 
in tho very popular incorrect response C. Pupils apparently 
loarned the applicable relationship: rate times time equals 

distance. Sound at the rate of 1200 ft. per sec. would travel 
3600 feet in 3 seconds. But the problem discusses an echo and 
the pupils failed to recognize that an echo requires that sound 
travels to a certain point and then returns. 



liiliiiiliil 






I 

I 



’ 










13 . 



18. 



25 . 
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TABLE k 



Book 1 Itoras fi-'om the Pretest Ivl8.stered by Fewer than 
2$fo of the Students on the Post test 



■ i.ti.iiMW«ir fuMp 



To find the scale of a model boat, you would 

A. 



B. 



•JC- C. 

D. 

E. 



find the difference between the length of thc5 model 
boat and the length of the real boat, 
measure both the length of the mast and the length 
of the sail since at least tvjo measurements are 
alviays needed. 

divide the length of the sail on tlie model boat by 
the length of the sail on the real boat, 
multiply the length of the model boat by the length 
of the real boat. 

divide the length of the real boat by the length of 
the model boat. 



pre 

post 



A 


B 


C 


D 


E 


CO 

C\J 

• 


.19 


• 

o 

CO 


• 

.12 


.27 


.36 


.10 


.10 


.09 


.33 



A lamp is 10 feet away. The light seems dim, so you move to 
a chair 5 feet away from the lamp. The brightness of the 
light is now 



A. 

B. 

C. 



5 times as much. 

10 times as much, 
tvjo times as much. 



*«* D • 


four times as 


much. 






A 


B 


C 


D 


pre 


.42 


.03 


.53 


.01 


post 


.26 


.02 


.70 


.01 


A hunter firest his rifle near a cliff. He 
of his shot 3 seconds later. How far awa.y 
from the cliff? (Sound travels at about 1 , 
second. ) 


A. 

B. 

C. 

D. 


1,800 feet. 
2,400 feet. 
3,600 feet. 

7,200 feet. 








A 


B 


c 


D 


pre 


.10 


.03 


.76 


, 08 


post 


.14 


.03 


CO 

• 


.04 













\ 

I 






s 



I 



\ 
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General Knowledge Items ; Table 5 

Those items were questions about astronomy topics which 
wore not part of the ESSP materials or, if they were present in 
the book, wore incidental to what was being taught. The questions 
wore included, as indicated above, to test the notion that this 
approach to the study of astronomy would also result in a con- 
comittant increase in students* general knowledge of astronomy 
oven though such information was not taught in Book 1. Nearly 
half (6 of 13) of the general knowledge items revealed a sig- 
nificant shift from protest to posttest: #3, #i?> #20, #23, 

and #24. As can be seen by inspecting Table ?, the items cover 
a range of topics and only throe could be considered to indicate 
mastery: #5 ( 76 % of the pupils answered it correctly on the 

posttost), #17 (73^ correct), and #23 (80^ correct). And all 
of these three were answered correctly by more than half of the 
pupils on the pretest: 53 of 92 knew #5, 4® knew #17, and 55 

know #23. 

The findings then indicate a significant trend toward 
increased knowledge of general information about astronomy, but 
do not reveal mastery of the information measured by the items 
from pro to posttest. It was not the intent of the ESSP 
materials to teach this general knowledge, so this aspect of 
assessment of these materials is not necessarily concerned 
with mastery. That a significant trend does exist showing 
an increase in the pupils* general knowledge of astronomy is the 
important finding. 








|W||||||||^^ 












mmmmm. 



- 23 - 

TABLE $ 



Goncrsil Knov/lods® Itioitis vjl*blo. Si gni. 1 * 10 suit Cixi Scju 8 ,pg Test 
of Change: Proportions of Responses by Students 



Drawing for Questions I to 

W. 



© 




^ /\vA/7 



© 




'V 



/ 



z' 



V. 






• 




IVhen the 


moon is at position' Z, there could be 


an eclipse of the 


A. 

•X- B. 

C. 


earth 

moon 

sim 

A B 


c 


t 


* 


pre 


.13 .23 


.6? 


4.0000 


p« . 05 


post 


.14 .36 


.52 






The small particles at 
the earth’s atmosphere. 


W in the drawing are 
They are probably 


falling through 


•:c- A. 

B. 
C • 

D. 

E. 


meteorites 

comets 

stars 

galaxies 

asteroids 










A B 


C D 


E 




pre 


.66 .09 


• 

0 

• 

0 


.13 


4.166 psr .05 


post 


.76 .14 


0 

• 

0 

• 


.03 






I 



I 



I 
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TABI;G 5 (cont’d) 



17. \Vhich of the following lists contains on ly planets? 



•J5- 



A. 

B. 

C. 

D. 



Neptune, Pluto, Uranus, Mercury. 
Jupiter, Venus, Sputnik, Earth. 
Earth, Mars, Moon, Jupiter. 
Uranus, Saturn, Sun, Phohos. 





A 


B 


C 


P 


pre 


.56 


.01 


.33 


.09 


post 


.73 


.01 


.23 


.03 



9.78 .005>p>.001 



20. Which list of planets is in the correct order of increasing 
distance from the sun? 



i’t 



A. 

B. 

C. 

B. 



Mars, Earth, Venus, Jupiter. 
Venus, earth. Mars, Jupiter., 
Earth, Ivlars, Jupiter, Venus, 
liars, Jupiter, Earth, Venus. 





A 


B 


C 


D 


pre 


.25 


.40 


.24 


.11 


post 


.20 


.59 


.11 


.11 


23. Which 


of the 


following is the 


best 



9.322 .005>p>.001 



A. 

B. 

C. 

D. 



Wandering stars. 

Groupings of stars. 
Non-moving stars.. 

Lines v;hich connect stars. 



B 



D 



.•2 



pre 


.0? 


.65 


.05 


.20 


8,166 


p= ,005 


post 


.0? 


• 

CO 

0 


.02 


.10 







24. '/i/hich list of objects is in the order of increasing size? 



A. 

B. 

*C. 

D. 



Solar system. Sun, Moon, Earth, 
Earth, Moon, Sun, Solar system. 
Moon, Earth, Sun, Solar system. 
Sun, Moon, Earth, Solar system. 



B 



D 



pre 


.17 


.12 


.53 


.14 


5.761 


pt= .025 


post 


.13 


.11 


.65 


.11 


► 














Book 1 Items on Posttost only ; Table 6 
There were nine Book 1 items on the post test which were not 
included on the pretest. They were only included on the post- 
tost because it was believed that the pupils did not possess 
the knowledge and understanding required by these items prior • 
to studying the ESSP materials. Pupil mastery ^^)as not achieved 
on any of the five items which required recall of factual 
knowledge: #29, #31, # 32 , #33, and #37. Two items in fact 

wore answered correctly by fewer than Z^o of the group (#32 
and #37). This may be a reflection of the authors* intent to 
place ’’groat weight on a few fundamental concepts of astronomy 
rather than on a scattering of isolated facts.” (Teacher* s Guide, 
page iii) 

Three items, #30, #3?, and #36, required the pupils to 
apply a rule or principle learned with Book 1 materials. While 



mastery was not achieved on any of these items, none fell below 



the Z^io level. One item, #34> requires knowledge of specific 
facts (Eratosthenes experiment) but also requires understanding 
of the underlying principles which organize the facts in order 



to apply then to a new situation. Pupil performance on this 
item approaches mastery ( 67 ^ correct). 

Grouping these nine items into three classes: recall of 

facts, application of principles, and knowledge of facts and 
organizing principles, from a lower to higher order of cognitive 
process involved, there also emerges a lower to higher pattern 
of pupil performance. Inasmuch as the authors* purpose was to 














TABLE 6' 

Book 1 Items Vililch Were on the Post test Only: Proportions 

of Responses by Students 



29. As you move an object avmy, it appears to become smaller because 
the 



A. apparent angle increases. . 

•Jc* B. angular size decreases. , 

C. atmosphere is hazy. 

D. earth’ s surface is curved. 

E. light has to travel further. 

ABODE 
.21 .47 .02 .19 .12 



I 

[ 30. You launch a balloon which is 12 feet in diameter and watch it 

\ rise. You then hold up a one foot ruler 2 feet in front of 

I you. The ruler just covers the balloon from one edge to the 

I other edge. How far away is the balloon? 




A. 6 feet 

B. 12 feet 
•3I* C. 24 feet 

D. 30 feet 

E. 48. feet 

ABODE 



i .44 .21 .25 .05 .03 

E 

? 

! 31. When Kepler made up the name Astronomical Unit it stood for 

I 

i A. the diameter of the earths orbit, 

i B. 186,000,000 miles. 

J 0, the distance the earth travels in one year. 

I D. the length of time it takes light to travel from 

I the sun to the earth. 

I E. the distance from the sun to the earth. 



s 



I 
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TABLE 6 (coat’d) 




32. To find the distance to the sun astronomers did not use the 
principle of the range finder because 



•J 5 - A. the angles are too difficult to measure. 

B. ' the sun is too big. 

C. it is impossible to direct a range finder at the sun. 

D. the earth moves around the sun. 

A BCD 



.23 .10 .30 .36 



33. To locate the orbit of an inner planet correctly on a scale 

model of the solar system, astronomers need to find the planet’s 

A. distance from the earth. 

'«-B. maximum angular separation from the sun. 

C. maximum distance from the sun. 

D. size and average length of day. 

ABC D 

.20 . 1^-9 .20 .10 



Tom lives in Miami, Florida, and Jerry lives in Tallahassee. One 
sunny afternoon Tom noticed that the telephone poles cast no shadows. 
He ran into the house and called up Jerry. Tom asked Jerry to measure 
the angle of the shadow of a telephone pole near his house. Jerry 
told Tom that the angle was 6 degrees. Tom looked at a map. and 
found that the distance between Miami and Tallahassee is 4.IO miles. 







I 

s 

.a 



I 



I 




.9 



I 



34. Tom wants to repeat Eratosthenes’ calculation for getting 

the circumference of the earth. Can Tom make the calculation 
with the information he has? 




D. 



No. 

Yes. 

No. 

but 

Yes. 



The calculation works only for Asvjan and Alexandria. 

The calculation v/orks any;vhere on the earth. 

The calculation was good for the ancient Greeks 

is not practical nowadays. ^ • 

If Tom also finds out the distance from Miami 



to Asvjan. 



I 

I 



I 
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TABLK 6 (cont'd) 



I 



E. Ho. One of the angles has to be measured in a well, 
ABODE 

.oi(. .67 .10 .03 .15 



35 . 



If Tom did make Eratosthenes* calculation with his data, 
what value would he get for the circumference of the earth? 



A. 

B. 

C. 

D. 

E. 



2 , 4^0 miles 
12,500 miles 
24,600 miles 

25.000 miles 

41.000 miles 



A 



B 



C 



D 



E 



.16 



.10 



.39 



.17 



.12 



36, If the diameter of a circle is 2 feet, its circumference is about 



A. 

B. 

C. 

D. 



E. 



If feet 
1,5 feet 
3 feet 
3.14 feet; 
6 feet 



B 



D 



E 



.14 



.09 



.10 



.15 



.50 



, / 

37. We divide a circle into 36O degrees because 



A. 

•X- B. 

C. 

D. 

E. 



a straight line must bo a I80 degree angle, 
people agreed to do it that vjay, 
an early Greek measured and found there vjere 36O, 
there is one degree for each section of the sky, 
360 is a universal constant. 



B 



D 



E 



.48 



.12 



.13 



.12 



.13 













I 



'h 



i( 
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focus on larger concepts rather than on isolated facts, the 
pupils* performance reflects success with the material in the 
direction intended by those who designed the materials* 



Skills Items : Table 2 

The last five items on the posttest were designed to test 
whether the pupils had acquired certain skills in making measure- 
ments, skills used by astronomers in learning about the universe. 

Each item was scored by two judges working independently 
and using a four point scales 0, 1, 2, and 3* Pupils who did 
not even attempt an item were given 0, so that in effect a 0 
score indicates an omission, A score of 3 indicates a complete 
and precise measurement. The other ranks on the scale, 1 and 2, 
were given to answers which were incomplete or showed less 
precise measurements than the criterion set for a rating of 3. 

The amount of error acceptable for each rating was decided 
upon beforehand, and the two raters agreed in all but a few 
cases which were then reviewed by the raters together. 

Items # 38 , #39, and #Ii.2 required the pupils to make some 
simple measurements using a ruler, protractor, and ruler and 
protractor respectively. Many pupils, 83 to 90?^, got a 
rating of 2 or 3 on those items, (See Table 8) The pupils 
in this study seem to be able to use these tools effectively. 
Pupil performance on items #40 &od #4l, however, was not 
comparable, those being mastered by only 1^3% and 2G% of the 
population. These two items required computation beyond the 
simple measurement, and this additional skill was apparently not 
learned by the pupils from using the ESSP materials. 
























% 
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TABD^ 7 



Skills Items 



38. 



Measure the length of the line segment below with your ruler. 
Write the best value for the length in the answer box. 




Answer 



39 . 



Measure angle A and angle B with your protractor. Write the 
values in the answer spaces. 




Angle A 



Angle B 



ll-O. 



The drawing below is a scale drawing made with a range finder 
to find the distance to the tree. The base line of the range 
finder A-B is 1.5 feet long. Measure with your ruler and then 
calculate the actual distance to the tree. Record your answer 
in the space provided. 




Distance to tree 










'k. 
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TABLE 7 (cont’d) 



Bill wanted to find the length of a truck which was 30 feet 
away. He held a matchstick near his eye so that it just^ 
covered the truck from front to rear. Belovj is a full size 
drawing of V 7 hat Bill saw. Make the correct measurements 
and calculate the length of the truck. Write your answer 
in the space provided. 




Length of truck 



' f 
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TABU!) 7 (cent ’cl) 



I {>2 « 



Using the 
RST. Use 



lino segment as one side, 
your ruler and protractor. 



draw a triangle 




L 






o 

ERIC 

iMHiilill 
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TABLE 8: Proportion of Students Rated at Each Level for 

Skills Items (N = 92) 



ITEM # 


omit 


1 


2 


3 


Mean 

Score 


38 


0. 


.03 


.36 


.61 


2.58 (92) 


39 


0. 


.10 


.20 


.71 


2.61 ( 92 ) 


40 


.07 


.51 


.09 


.34 


1.81 (86) 


41 


.21 


.54 


.07 


.19 


1.55 (73) 


42 


.04 


.12 


.26 


.58 


2.48 (88) 



* Computed on the basis of pupils who gave some response. 
Number of cases in parentheses. 






B. SPPSC'TS ON STUD-']NrS OP Tin 



,34 • 

\-:sF :/ T3Hi-\LS 



Test On Uadcr&tandig.f^ dcicne. 

The third hypothesis propose.. cl o. ncorncd nn incro'^sc in 
** 5 tudcnts* gcgcrril undcrst^'^iidiiig Cif science, (cis Tuensured by 



the T^st On Understanding Scionco)* To test this hypothesis, n 



t for correlated groups was computed fro'ii the scores of the 
TOUS protest and posttest, (isec Table 9.) 



The co'iputed value of ^ indicates a significant difference 
at tho .0^ level between TOUS pre and post test means. But 
the actual difference between tho moans is only .88, a moan 
change of less than one item correct from pro to posttest. 
Closer inspection of tho data seemed a possible source of more 
meaningful information about what happ'^ed from pro to posttost 
administration and about which items contributed most to this 
change in moan score. 

Again, to employ the McNomar chi square test of change, 
contingency tables were formed for the 38 items on tho TOUS 



tost. Those items which revealed a significant shift are 
shown in Table 10. 

Seventy percent of the pupils having chosen the "best" 
answer was used as an index here, parallel to the ']0% level 
of mastery used in tho analyses of the subject-matter achieve- 
ment tost items. Of tho six items with significant chi squares, 
on only throe was the best answer chosen by 70^ more of 
the pupils on the posttost. (#5, #12, and #26.) And it is of 
interest to note that these three "best" answers were known by 
more than $0^ of the pupils on tho pretest, the best answer 
to #12 oven having been chosen by almost 70 %, 




f 

I 
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TABLE 9 



Comparison of Pretest and Posttest Scores on TOUS 
^ (N = 89) 





Mean 


S.D. 


t 


Pretest 


19.93 


U.iii 


2.01 p = .0? 


Posttest 


20.80 


It. 399 









HHI 
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TABnu 10 



TOUS ItemwS with Significant Chi Square Test of Change: 
Proportions of Responses by Students 



5 . 



i‘\/hl ch 


of the ; 


following 


5 sentences about science is best? 


A. 


Modern 


science 


is too advanced to use past discoveries. 


B. 


Modern 


science 


develops 


modern products. 


c. 


Modern 


science 


depends 


on useful inventions. 


'X- D. 


Modern 


science 


is based 


on the science of the past. 




A 


B 


c 


D 


pre 


.07 


.26 


.14 


.^2 9.142 , 005 >p >.001 


post 


.03 


.16 


oil 


.70 



i 

i i 



6 , A scientific theory should 



A. 

B. 

•X- C. 

D. 



provide the final solution to scientific problems, 
suggest directions for making useful things, 
tie together and explain many natural events, 
suggest good rules for carrying out experiments. 



A 



B 



D 



pre 


.23 


0 

CM 

• 


CO 

• 


.18 


post 


.13 


.20 


.?o 


.17 


12. The 


scientists of today 


can 


work on 



1|.000 



p= .05 



A 

B 

C 

D 



work harder than earlier scientists, 
have more ideas than earlier scientists, 
build on the vjork of earlier scientists, 
are more clever than earlier scientists. 



B 



D 



pre 


.01 


CM 

• 


.69 


.05 


post 


.01 


.1? 


.79 


.04 



4.166 



p= .05 



A • 

B. 

C. 

D. 





A 


B 


0 


D 


pre 




.16 


.19 


.12 


post 


.36 


ro 


.21 


.16 



24. VJhon a scientist makes a new discovery, he usually makes a 
report of it because he 
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TABLE 10 (cont’d) 

VJhich of the follov;ing is the main need of science? 



People V7ith new ideas. 

More money and equipment. 
VJoll- trained craftsmen. 
Better v/orking conditions. 



5.444 



p.= .02 



A. hopes to help mankind by announding his discovery. 

B, wants to prevent other scientists from making the same 
discovery, 

■jc* C, wants other scientists to know about his work and check 
it. 

D, hopes other scientists will help him to finish his work. 

2 





A 


B 


C 


D 


pre 


, 66 


0. 


.26 


.09 


post 


.34 


.03 




.11 



X‘ 

16.030 



p y , 001 



26, Before a scientist announces a new theory to the public, he 
will most likely talk his ideas over vjith 



government leaders who many vjant to use his theory, 
other scientists in his special field, 
science writers of large newspapers, 
a group of experts on scientific theories. 





A 


B 


C 


D 




pre 


.13 


.59 


0. 


• 

ro 


8.333 


post 


.08 


.77 


.04 


.11 





* p > . 005 



1 



\ % 
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Two of the remaining three items, #6 and #24, wore known 
by approximately half of the pupils on the posttest 
^2fo respectively). And the last of the answers which showed 
a significant chi square for change ( #17 ) shifted in the 
negative direction. Fifty-two percent of the pupils chose the 
best answer on the pretest, while only ^6% chose it on the 
posttest. 

A second kind of closer look at TOUS test results was an 
examination of individual students* pretest to posttest changes. 
Of the 89 students for whom complete pre and posttest data on 
the TOUS are available, 31 achieved lower scores on the 
posttest than they had on the pretest. This drop ranged from 
-1 to -9, the moan loss being -3.I4.2. In 53 cases there was 
an increase from 1 to 13 points, the moan gain being 3.82, 
and three pupils’ TOUS scores were the same on both the pretest 
and posttest. 

Prom the preceding evidence, it is difficult to find support 
for our hypothesis that study of ESSP materials will ’’increase 
students’ general understanding of science (as measured by the 
Test On Understanding Science ).” The t test indicates a 
statistically significant (.05 level) difference between the 
pretest moan and the posttest mean, but on very few posttest 
items (3) did lOfo or more of the pupils choose the best answer, 
and a largo percentage of the population (37.1^) appeared to 
have decreased ’’general understanding of science” as measured 
by this instrument. 




Our fourth hypothesis concerned ’’general understanding of 



1 






i- 



ERIC 




science” also. It posited that ”a student’s gain in general 
understanding of science is positively correlated with subject 
matter achievement,” To test this idea, two coefficients were 
computed to examine the correlation between the pro to posttost 
differences on the subject-matter test with the pre to posttest 
differences on TOUS. As can be seen from Table 11 , two 
’’difference scores” were calculated for the subject-matter tests. 
One was the pre to posttost difference on Book 1 items, the l5 
questions which appeared on both tests, (This difference, there- 
fore, does not include the 9 Book 1 items which appeared on the 
posttest only,) The other subject-matter pre to posttost 
difference which was calculated was the difference between the 
total pretest score (28 items including Book 1 and General 
Knowledge questions) and the total posttest score ( 1|2 items, 
consisting of the 26 pretest items, an additional 9 Book 1 
items, and 5 skills items). In this way we hoped to see whether 
gains in achievement on ESSP materials was correlated with 
increase in general understanding of science, and also whether 
the results of the total experience, as measured by the post- 
test, with the achievement on the pretest subtracted out, would 
bo related to gains on TOUS, In the table the former difference 
is labelled ’’Book 1 difference” and the latter is ’’Total Test 



difference , ” 

VJhile our earlier discussion indicates that little evidence 



can be found for real differences in TOUS scores from pre to 
posttest administration, we were still interested to know 












— 1^0 “ 
TABLE 11 






1 



Correlations Between Pretest to Posttost Differences 



TOUS Difference = .876 



Confidence 

Correlation Interval 



Book 1 

Difference = I .876 



.386 .95 (. 196 ^ . 618 ) 



Total Test 
Difference = 8,685 



.198 .9$ (. 010 ^ .kl2) 



I 

I 

i 



I 



1 

I 



I 

‘i 

I 



I 

I 



I 



I 



I 



"i 

I 









I: 

I 

’i 



I 



1 

I 
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whether what change we did find was at all related to subject- | 

matter achievement. The indices generated by our correlation of I 

test score differences does not seem to support the hypothesis. I 

Quite the contrary, it seems almost a chance occurence that | 

more pupils gained than lost from pre to posttest on the TOUS. | 

I 

I 

Word Association Study | 

The semantic differential inatr\iment, ’’Word Association | 

Study” (WAS), was designed to test our fifth hypothesis, viz., | 

j 

’’Study of ESSP materials will affect students* views of astronomy, | 

I 

7> 

arithmetic, scientists, and learning experiences in science." I 

I 

The WAS instrument, as previously described in this report, j 

i 

yielded pupils* ratings of eight concepts on each of l5 bipolar | 

I 

adlectival scales. Thus, the data collected for testing our i 

“ J 

fifth hypothesis consisted of 120 pairs of ratings from the | 

I 

pre and posttest administration of the WAS instrument. | 

(In the WAS instrument, ratings of 1 to 5 were assigned j 

to the five positions on each scale. This procedure assumes j 

I 

an equality of the intervals between scale positions. Although j 

I 

this assumption was not tested in this study, there is support | 

I 

for it from previous research with the semantic differential, 



and our procedure of assigning equal interval ratings is com- 
monly used. Furthermore, some of the scales on the WAS instru- 
ment are ’’reversed” as printed in the test booklet to counter- 
act a possible response set on the part of the student. The 
ratings of these ’’reversed” scales are converted during the 



analysis so that the numerical values of scales with similar 








I 




meanings will bo consistent. Thus, for example, the fifth 
scale as printed on the WAS instrument is "important-unimportant", 
but this "reversed" scale is converted during the analysis into 
an "unimportant-important" scale with a rating of 1 assigned 

to the "unimportant" polo and ratings of 2, 3» ^ 

assigned to the other scale positions so that the order is 

consistent with increasing "importance.") 

The moans and standard deviations of the pretost and post- 
test ratings were calculated for the 120 items on tho WAS 
instrument, and a representative selection of these is pre- 
sented in Table 12. The table also shows the changes in mean 
ratings of the several scales from pre to posttost, and we 
may take note of a number of these shifts that are rather 
suggestive. In commenting on these shifts, we are aware that 
only limited confidence can properly bo placed in the ratings 
for any one scale on a semantic differential instrument; hence 
wo treat tho changes in moan ratings as suggestive, rather 
than as strong evidence of changes in students' perceptions. 

Wo estimate that, generally, a change in mean rating for an 
item in excess of 0.20 is statistically significant at the .0$ 
level, and that a change in excess of 0.26 is statistically 
significant at the .01 level. (This estimate is based on 
calculations of a t for correlated means for a random sample 
of ton items in the table.) On this basis, then, we note in 



the data presented in Table 12 that: 

=?i-,idents seemed to view ASTRONOMf as loss powerful, loss 
excitiS ani less enjoyable on the posttest than on *he pretest. 
P?om pro to posttost, they came to regard it as loso diffic , 



I 
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TABLE 12 



Pretest to Posttest Chs.nges in Mean Ratings 
Concepts and Scales on the iVord Association 
(Ivlinimum rating r: 1 ; maximum = 5 ) 



of Selected 
Study (N:r 92 ) 



I: 



1 

i 




Pretest 


Post test 


Pretest to Posttof 


1 


Concepts 
and "Scales 


Mean 


S.D. 


Mean 




Change 
in "Mean 


ft 

1 

1 


ASTRONOMY 












1 

1 


we a k-p ovje rf ul 


4.054 


0. 864 


3.739 


0 . 924 


-0,315 


f 

1 


hard- soft 


2,413 


0.963 


2.565 


0.856 


+0.152 




unimportant- important 


4.628 


0,524 


4.587 


0 . 854 


-0,241 I 




dull-exciting 


4.602 


0.609 


3.815 


1 . 240 


-0.787 ' 


% 


mann i s h- w oma n i sh 


2.359 


0.897 


2,522 


0,733 


+0.163 


h 


unenjoyable-en joyable 


4.355 


0.917 


3.924 


1,188 


-0,431 


l 


changing-permanent 


1.796 


1.119 


1,804 


1.008 


+0.008 




easy-difficult 


3,806 


0.912 


3.543 


0,907 


- 0.263 


1 


ARITHIvIETIC 

weak-powerful 


4,000 


0,921 


• 

4.011 


1,093 


+ 0,011 


I' 


hard-soft 


2,473 


0.951 


2 , 700 


0.893 


+0,227 


S''' 


unimportant-important 


4.957 


0.204 


4,856 


0, 628 


- 0,101 


1 

r 


dull-exciting 


3.710 


1.247 


3.856 


1.286 


+0,146 


? 


manni sh-womani sh 


2.925 


0 . 448 


2.911 


0.630 


-0.014 


P 

1 ' 


unenjoyable-en joyable 


3.660 


1 . 1 d 6 


4,022 


1.281 


+ 0.162 


1 1 


changing-permanent 


2.355 


1.486 


2.644 


1,352 


+0.289 


•** 

1 

* 


easy-difficult 


3.247 


1.204 


2.911 


1.088 


-0,336 


> 

1 

i 


PiEADING ABOUT SCIENCE 
slow- quick 


3.000 


I.O63 


3.130 


1.121 


+0,130 


t 

V 


hard- soft 


3.032 


0,865 


2,890 


0,781 


-0,142 


i 


unimport ant- import ant 


4.796 


0.523 


4.620 


0,850 


-0,176 


\ 

t 


unenj oyable -enjoyable 


4. 152 


1,026 


3.783 


1.184 


-0,369 


1 

1 " 


use less -useful 


4.753 


0,564 


4.511 


0.978 


-0,242 


1 

t 

1 


easy-difficult 


2.763 


, 1.026 


2,652 


0.999 


- 0,111 


1 

1 

1 


MAKING IvISASUREIvlENTS 
slow- quick 


3.054 


1.087 


3 . 043 


1,128 


- 0.011 


f’ 


hard-soft 


2.828 


0,880 


2,707 


0,846 


- 0.121 


1 

?k 


unimport ant -import ant 


4.591 


0.741 


4 . 641 


0.673 


+0.050 


?r* 

t 

k' 


unenjoyable-en joyable 


3.828 


1,185 


3.761 


1.152- 


-0.067 


n- 

1 


useless-useful 


4. 667 


0.727 


4.565 


0.789 


- 0,102 


f 


easy-difficult 


2,968 


0.983 


2.902 


0.973 


- 0,066 


I 


MOST SCIENTISTS 
weak-powerful 


4.097 


0.873 


3.891 


0,931 


- 0,206 


j. 


dirty-clean 


4.183 


1.032 


3.891 


1,104 


-0.292 


t. 


dull-exciting 


4.376 


0,932 


4.065 


1.003 


-0.311 


i 

i' 


manni sh-womani sh 


2.409 


0,837 


2.489 


0.832 


-O.OSO 


1 


moving- still 


1 . 914 


0.974 


2.141 


1 , 044 


+0.227 


'f 


j useless-useful 


4.871 


0,396 


4. 652 


0,907 


-0.219 


r 


f oolish-wisc 


4.710 


0,600 


4.641 


0,793 


-0.069 




boring- interesting 


4.527 


0 . 7O8 


4,087 


1.096 


-0,440 






I 
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TABWi; 12 (cont'd) 



Oonce.pt s 

f«i»*«IW*«''W«f»*«» *««•>« ♦•*•»♦♦ *iW 

and .Scales 



m SCIENCE TEACHER 



we a k- p 0 vj o r* f \il 
dirty- do an 
dull-oxoit 7 lng^ 
ma n n i sh - w o iin n 1 s h 



moving- still 
useless-useful 
foolisb.-v 7 ise 
boring- into re sting 



Pretest 
Mean S.D, 



3.871 0.923 

k . k9y 0.916 

4.301 0.906 

l|-. 55'9 0 . d$3 

l . 62 )|, 0.793 

4.699 0.749 

4.667 0.771 

4.4.62 0.854 



Posttest 
Mean ‘S.D. 



3.253 1.131 

4.000 1.211 

3.692 1.347 

4.527 1.015 

2.297 1.206 

4.407 1.164 

4.187 1.26k 

3.802 1.368 



Prctc'st to Po.'sttoat 

» j jj-iJ.jruL«.-fiirTniiirr tTxnuniri^r— rr* — y— ‘ »»v *» pwi 



Cb 


.anno 

nji III t Will** wr 


• 

in 


iuoan 


- 0 . 


618 


- 0 . 


495 


- 0 . 


609 


- 0 . 


032 


+0 • 


673 


- 0 . 


292 


- 0 . 


400 


- 0 . 


660 



o 
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and thoro was no shift in thoir viuw of astronomy as changing | 

or permanent. t. 

I 

Between pretest and posttest, the pupils shifted in their | 

view of ARITHI®TIC toward seeing it as less difficult. It | 

also seemed to become ’’softer” for them and more permanent, | 

I 

I 

READING ABOUT SCIENCE was viewed as less enjoyable by the | 

students on the posttest than on the protest, i 

I 

The pupils’ view of MOST SCIENTISTS seemed to shift from | 

pro to posttost toward loss clean, less exciting, and loss in- j 

torosting. The shifts in the view of MOST SCIENTISTS are | 

fairly closely matched, though to a loss degree on each scale, | 

with the shifts in the view of IvlY SCIENCE TEACHER, | 

The SCIENCE TEACHER appears to have suffered some losses j 

in the pupils* view from pretest to posttest. On the second | 

occasion, she was seen as less powerful, loss clean, less 
exciting, less useful, loss wise, loss interesting, and more 
still. 

While changes such as these in moan ratings on single 
scales for particular concepts are suggestive, further analysis 
of the responses to the WAS instrument provided information 
of somewhat greater interest about the pupils* perceptions. 

The response data generated from the administration of a seman- 
tic differential instrument lend themselves admirably to factor 
analytical techniques for the purpose of uncovering the under- 
lying structure represented by the responses. The aim of the 
factor analysis is to discern common factors made up of scales 
on the semantic differential for which pupils* responses tend 
to cluster together. To use Osgood’s terminology, these 
factors represent dimensions of the pupils* meaning space. 

We prefer to think of these factors as constitutont elements 
of the pupils* images of particular concepts. The elements 
of the image of a concept may bo cither cognitive or 
attitudinal. The -nature of the scales selected for a given 
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somantic difforontial instrument determines whether cognitive 
elements, attitudinal elements, or both, may bo discornod in 
the analysis of the responses on the instrument. For the WAS 
instrument, we selected adjectival scales that were primarily 
attitudinal, so that wo would expect the factor analyses to 
reveal some of the attitudinal elements of the pupils’ images 
of the concepts. 

The computer program which wo used performed a principal 
components factor analysis of the intercorrelation matrix and, 
when called for, rotated factor analysis was made of the l5 
scales of the WAS instrument across all eight concepts. This 
analysis produced one (unrotated) factor that accounted for 
65.8^ of the variance. Scales with high factor loadings on 
this factor included; unimportant-important,, dull-exciting, 
bad-good, unen joyable-onjoyable, useless-useful, foolish-wise, 
boring- interesting. Further preliminary analyses showed 
evidence of considerable interaction between scales and partic- 
ular concepts, indicating that there was no single factor 
structure common to all the concepts rated on the WAS instrument. 
Hence, wo subsequently factor analyzed separately each concept 
(or page) of the WAS instrument across the 1$ scales on which • 
the concept was rated. These analyses showed that there were 
three WAS instrument concepts for which the factor structure 
was very nearly the same. 

Table 13 presents three principal (rotated) factors that 
our factor analyses revealed for the concepts ASTRONOMY, IViAKING 
IVEASUREIiffiNTS, and DOING SCIENCE EXPExRIMENTS . The cumulative 
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TABLE 13 



Three Principal Factors and Percent of Variance 
for Three Concepts on the V/ord Association Study 



CONCEPT: 



ASTRONOMY 



MAKING 

i\flEASUREiviENTS 



DOING SCIENCE 
EXPERIlvIENTS 



Percent of Variance {$ rotated factors) 



Factor I 



PERSONAL ENJOYi^IENT 
dull- exciting 
bad-good 

unenjoyable-en joyable 
boring- interesting 



17.2 



19.0 



23.0 



Factor II IMPORTANCE 



13-0 



12,2 



16.8 



unimportant- important 

useless-useful 

foolish-wise 



Factor III 



DYNAMISM 
slow- quick 
weak-powerful 



12.1 



9.6 



13.1 



I 



i 
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porcontago of tho total variance accounted for by theso three 
factors typically ranges between 1^0% and $2fo, (Tne percentages 
shown in tho table are based on pretest data for ASTRONOlvIY and 
MKING i®ASUREl®MTS, and on posttost data for DOING SCIENCE 
EXPERIiviENTS . ) The table also shows the scales that had high 
factor loadings on each of tho three factors, and it was from 
the meanings associated with the included scales that wo 
constructed the names assigned to each factor. Factor I in- 
cludes scales that seem to represent a child’s personal 
involvement; tho clue hero, wo believe, is that a fifth- grade 
child generally talks of something as either ’’exciting” or 
’’bad” or ’’good” in terms of his own experiences, rather than 
in an abstract sense. Hence, we gave the name ’’Personal 
Enjoyment” to Factor I. The scales included under Factor II, 
on tho other hand, do not seem to reflect this same personal 
involvement, but point to a more detached and somewhat more 
sophisticated evaluation of concepts on the part of these 
fifth graders. In fact. Factor II appears to have a close kin- 
ship with the ’’Evaluative” factor repeatedly found in semantic 
differential studies with adults as subjects. We chose tho 
name ’’Importance” for Factor II. Consideration of the two 
scales included under Factor III suggested that ’’Dynamism” 
would be an appropriately descriptive name for this factor. 

Wo have, then, throe constituent elements of the pupils’ images 
of throe concepts which they rated on tho WAS instrument, 
elements that denote the personal enjoyment j. importance, and 
dynamism of these concepts as the pupils viewed them. 



j 

! 
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What wore tho oUangos, if any, in tho pupils' views of 
those concepts during tho time that they studied the ESSP I 

matorials? To answer this question, we computed a composite j 

I 

score for each principal factor identified for each of tho J 

concepts, ASTRONOMY, MAKING MEASUREMENTS, and DOING SCIENCE 
EXPERIMENTS, from both tho pro and posttost data on the WAS 
instrument, Tho composite score for a factor that includes 
k scales is the sura of l/k times tho rating of each scale. Thus, 
for oxanple, tho composite score for "Importance" is equal to 
the sum of l/3 times the rating on the "unimportant-important" 
scale, 1/3 times tho rating on tho "useless-useful" scale, plus j 

1/3 times the rating on tho "foolish-wise" scale. We also | 



calculated the pro and posttest means and standard errors of 
the composite scores for "Personal Enjoyment," "Importance," 
and "Dynamism" on each concept and, using a jb“*tost for cor- 
related data, made comparisons of the nine protost-posttest 
means. (See Table 14 . ) 

As is displayed in the table, the students’ image of 
ASTRONOMY decreased in the element of "Personal Enjoyment" 
(change in mean composite score significant at the .01 level) 
and in the element of "Dynamism" (change in mean composite score 
also significant at the .01 level) during the time they wore 
studying the ESSP materials. With reference to their personal 
enjoyment of astronomy, the protest moan composite score for 
the group was extremely high (about I 4..5 out of a possible 
maximum of 5), and tho posttest mean composite score (about 4*0) 
was still very high. There was, however, a considerable spread- 
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TABHW 111 



Pretest-Posttcst Comparisons of Three Factor heans 
for Three Concepts on the V/ord Association Study 



DOING SCIBNCK 

ASTRONOi-ir KAKim ];lCASlTill!]iM13NTS EXPERIMENTS 



Fact 


:or 




Mean 


S.E, 


lie an 


S.E. 


Mean 


S.E. 


I - 


Enjoyment - 


Pre 


4.519 


0.067 


4.106 


0 . 084 


4.657 


0,050 




mm 


Post 


3.976 


0.110 


3.965 


0,102 


4.481 


0,073 




Change 




-0.543 


0.106 


-0.141 


0.107 


-0.176 


0.072 








(t= 


. ?.13**) 


(t. 


=1.32) 


(t= 


2.44*) 


II. 


“Importance 


“Pre 


4.584 


0.058 


4.557 


0,058 


4. 710 


0 , 040 






“Post 


4.473 


0,077 


4.41|-8 


0.074 


4.624 


0.064 




Change 




.-0,111 


0,073 


-0.109 


C.078 


-0,086 


0.062 








(t= 


^ 1.52) 


(t= 


1.40) 


(t= 


1.34) 


III. 


- Dynamism 


“Pre 


3.576 


0.075 


3.234 


0.084 


3.359 


0.075 






“Post 


3.272 


0.062 


3.261 


0.080 


3.408 


0,075 




Change 




-0.304 


0.080 


+0,027 


0.096 


+0.049 


0,090 








(t= 


3, 80-!f-;:- ) 


(t= 


r 0.28) 


(t= 


0.54) 






Significant at ,0^ level 
Significant at .01 level 
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ing out of tho group’s personal cnjoymont ratings of astronomy, 
as is shown in the increase in the standard error of the moan 
composite score from O.O 67 on tho protest to 0.110 on tho post- 
tost. With reference to the students’ imago of tho dynamism 
of astronomy, there was a statistically significant shift from 
pro to posttest, as already noted, toward a loss dynamic view 
of tho subject. 'Iho posttest moan composite score of about 3.3 
(between a possible range of 1 to indicates that, after study 
of the ESSP materials, the pupils’ view of astronomy included 
an olomont of only moderate dynamism. Lastly, tho students’ 
image of the element of "Importance” in their concept of 
astronomy did not change significantly from pro to posttest, 
and the means of the composite score on botn occasions (about 

4.6 and about 4.5) were extremely high. 

Turning to the concept MA.KING ivlEASUREIviENTS, no significant 
changes wore found in tho students’ view from pre to posttest. 
Referring to the element of personal enjoyment in their view of 
making measurements, both the pretest moan (about 4 .I) and the 
posttest moan (about 4 .O) of the composite score were very high. 
The importance of making measurements in science is certainly 
stressed in the SSSP materials, but this element in the students’ 
image of measurement was already extremely hign on the protest 
(moan composite score of about 4*^)5 that rather little group 
gain was possible. The slight decrease in tho mean composite 
score for "Importance" on the posttest is not statistically, or 
educationally, significant. With reference to the third factor 
on this concept, tho pupils’ image of making measurements 
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included an olomont of only modorato dynamism, judging by moans 
for the composite scores of about 3.2 on both pro and posttost. 

The element of "Importance” in the pupils’ view of DOING 

SCIENCE EXPERIIVIENTS showed extremely high means of composite 

scores, both on the pretest (about l4.«7) *ind on the posttost (about 

4 . 6 ), and the slight decrease in the moan on the posttest is 

not significant. For the same concept, the decrease from pro 

to posttost in the moan composite score for the element of 

"Personal Enjoyment" is statistically significant at the .05 

level. It may bo that the pupils’ use of the ESSP materials 

had some effect on this element of thor perception of doing 

science experiments, but we are not inclined to become wildly 

concerned about a loss of loss than 0.2 in moan composite score. 

We note that the element of enjoyment in the pupils’ view of 

high 

doing science experiments remained extremely/ throughout , with 
a moan composite score of about 4*7 and about 4«? on the pro and 
posttost, rospoctivcly. Finally, DOING SCIENCE EXPERIlvENTS was 
perceived not very dynamically by the students, and there was 
no significant change in this element of their view. 

Let us new see what bearing the foregoing analyses and dis- 
cussion have on our hypothesis for this part of the study 
(Hypothesis 5 ). Ne have presented findings which show that 
the pupils’ view of astronomy was changed during the time that 
they studied the ESSP materials, (See Table l4 and Table 12.) 

Wo have also presented some suggestive evidence of changes in 
the pupils’ views of arithmetic and of scientists. (Table 12). 

We have found little evidence in support of the idea that study 
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of ESSP materials will affoct students* views of learning experi- 
ences in science. In summary, our work with the WAS instrument 
provided partial support for the fifth hypothesis, that students’ 
views of astronomy, arithmetic, scientists, and learning experi- 
ences in science would be affected through their study of the 



ESSP materials. 
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CONCLUDING REivL^RKS 



This study has investigated the effectiveness of the ESSP 
luaterials in increasing students* knowledge of astronomy and of 
how astronomical information is obtained. We found that the 



grade students in the University of Chicago Laboratory 
Schools who studied the ESSP materials were moderately successful 
in mastering some of the topics that were taught. We also found 
gains in the students* knowledge about certain astronomical 
topics that were not specifically taught in the ESSP Book 1 
materials, and this was interpreted as an increase in the stu- 
dents* general knowledge of astronomy. Detailed analyses of 
subject-matter achievement test items were made to determine 
more precisely what the students did learn and did not learn in 

their study of the ESSP materials. 

The study also investigated the effect on students of study- 
ing the ESSP materials. We found that the effect of studying 
these materials on the students* general understanding of science 
was slight. The extent of this effect was explored through 
analyses of items on the Test On Understanding Science ^ Through 
the use of a semantic differential instrument, we investigated 
further the ESSP materials* effect on students* perceptions. 

We found that study of the ESSP materials did affect the students’ 
view of astronomy, probably affected the students* views of 
arithmetic and of scientists, but did not seem to affect the 



students* views of learning experiences in science. 

Perhaps more important than the specific findings of this 
study, however, are the procedures of analysis and interpretation 
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which are illustrated here, 'Jlie specific findings of the study 
provide some valuable information about the effectiveness and 
effects of the ESSP materials, but this information should not 
bo looked upon as a firm evaluative judgment of these materials 
unless and until the study is adequately replicated with other 
students and in other school settings. On the other hand, the 
procedures used in the study have wide applicability in the 
evaluation of innovations in curriculum materials. These pro- 
cedures provide specific information about what knowledge and 
which ideas students mastered successfully, about where they 
failed to attain mastery, and about • changes in students’ per- 
ceptions of the subject. The developer of new curriculum 
materials can learn a great deal from his evaluation efforts 
if he will make conscientious application of the analytical and 
interpretative procedures which are illustrated by the present 
study. 



AGKNOWLEDGEli/ENTS 

The investigator wishes to record his indebtedness to the 
persons who assisted in conducting this study and in preparing 
this report. Grateful thanks are e-^tended to Dr, J. Myron Atkin 
and staff members of the University of Illinois Elementary 
School Science Project, for supplying materials and for financial 
support; to Barbara Wehr for cheerfully undertaking the teaching 
of the ESSP materials; to the fifth grade Lab School students 
for their essential participation; and to ray research assistants. 










■ 55 ." 

Fred Geia, Jr., E. Lawrence Lisa, and Mary E. McOuUou^, for 
Invaluable help. Hie typing of this report was conscientiously 
and cheerfully performed by Miss Barbara Lee, whose fine work 
is greatly appreciated. 












wmm 



im 




ERRATA 



page 13. line 6 




using the normal d:istribv.i:.ioxi o3 
the stat5-stic 2'{H‘~4)s:, 



page 37s Table 10 

for item no<» 24? READ; 
for item no. 26., READ ; 

mi III— I—— mill 



P < .001 
P < -005 



page L6 , line 10 



HEAD 



when called for^ 
according to the 
Our first factor 
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e r 1 1 erion o 
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